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The paper addresses issues in special education 
program evaluation. Introductory information examines the mandate for 
state monitoring under P.L. 94-142, The Education For All Handicapped 
Children Act, and reviews state monitoring approaches undetrtaken in 
Nebraska, Missouri, and North Carolina. A nine-step procedure 
designed to help policymakers become more effective consumers of 
evaluation reports is provided. Checkpoints for policymakers are 
listed under each of the following steps: reading the complete 
evaluation report, reviewing the evaluation model, reviewing the 
evaluation's goals and objectives, reviewing the evaluation criteria, 
reviewing the population and sample selection, reviewing the 
evaluation design, reviewing the data collection and analysis 
process, reviewing the overall evaluation report, and designing a 
program improvement management plan. Also included is a glossary of 
approximately 40 evaluation research terms. (CD 
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O PWDOtWi EVALWTI Ot TOR lyPHC TI VE SPHCIAL EDUGCTION 

1 1 I 

^petMMl education progpams in the United States have expanded from state 
special schools for the deaf and blind founded in the late 1800s, to wll over 
15,000 stale-authorized public school programs in 1983. Thej§e programs serve 
over three million pupils annually at a cost of millions of dollars of federal, 
state and local monies. 

Over the last hundred years, evaluation of these programs has also expanded, 
fr^>m no evaluation of special education programs to intensive evaluations for 
t>oth local and state use. Improvement of the quality of educational programming 
fo r students the single most desirable outcome of evaluation . Program 
evaluation also helps assure accountability, and assists planning for budgetary, 
personnel, facility and materials needs. It provides local, slate and federal 
decisionmakers with critical information; it fosters cwmunicat ion among diverse 
audience;; it clarifies program object ives and aceompl ishments. 

Although there are nrwny journal articles and monographs about evaluation 
theory and practice^ there are few resources available to help policymakers be 
more thoughtful and effective ''consu&ners'' of the special education program 
evaluations they receive. This issue brief addresses this need through a focus 
on the following questions and their inplications for stale and local education 
pol K^ymakers: 

• m\T WPS SPRCIM. RDUXnON PHOCmM KVALUATfON MRAM? 

« iKW \HV snvT^: ST'VrKS HKLPlNtl IXTAL DISTRICTS TO ASSESS AW) I\f'«OVK TW 
QUMJTY (>P Kl)LrATI(>MAL PfOHiVVMINt; FOR H\NT)H>\PPFJ) OlfLDRES? 



• m\T AHF THE ESSENTIAl.S OF AN AlfiJUATE EVAUIAriON? 

. ........... v.. •. 
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Vmr IXWS SPICIAL RDUQ\TiON PIOaUM BVAUWriON MEAN? 



The {Wsm dftte for State Monitoring UiKter P>L> 94-142 

Tho intent of the monitoring provisions in P.L, 94''142 is to assure a sir^le 
line of nccotintabi H ty for the eduealton of all handicapped children within a 
state. The federnl statute thus provides a broad mandate for state education 
agencies (SFAs), riHjuiring that they: 

".••provide for procedures for evaluation at least 
annuH 11 y of the effectiveness of programs in meeting the 
edueationai needs of handicapped children (including 
evaluation of individualized educational progran»). In 
accordance wi t h such cr i ter ia that the Commiss ioner 
shall prescr it>e*..." (Section 613,(a)(ll)) 

This mandate is made more specific in the federal regulations: 

"\ general application must include assurances, satisfactory to the 
c^^rmi'^sioner. . , 

(3) That the state will adopt and use proper methods of a<tninister ing each 

program, i nc 1 ud i ng : 

i ) \tonitor ing of agencies, inst itut ions, and organizations 
responsible for carrying out each program, and the enforccsnent of 
any obligations im{K>sed on those agencies, institutions, and 

organizations under law; 

ii) Providif.^ technical assistance, if necessary, to those agencies, 
institutions, and organisations; 

jii) Kneouraging the adoption of promising or innovative educational 
techniques by those agencies, institutions, and organizations; 

iv) The dissemination throughout the state of information on program 
requireTwnts and successful practices; and 

v) The correction of deficiencies in program o{>er8tions that are 
identified throur^h moni tor i ng or evaluation.** (Rducat ion Division 
Oneral Acini nist rat ive Regulations, Section 100b* 101 (3) (3)) 

<=^ince PJ^. 94-142*s passage, *?KAs have implewnted monitorir^ syst^ns 
dr^ifjnpd to generate the informfition required by federal law and regulation. 
I rulrflyin^ this apprwch is an assunpt ion that conpliance with PpL^ 94-142*s 
specific requirements will assure quality programning. 

As sessir sf the Qual i ty of 1^[)ecial Edtocaitlon Progfww 

Recently, state nx?nitoring experiences have led irony %?A officials to argue 
that '^coinpl iance moni tor i ng efforts ^ at least as they have been eoftduc'ted In *he 
pa^t, have missed some fundamental, qualitative aspects of educt^^ioiwl prc^fams 
for handicapped children" (Farrow, 1983,8). For exanple, it is relatively easy 

(Continued on inside back cover) 



to obsf^rvf* vvtifttipr the necc^«sary procedures arc in place to develop an 
indiviiJiml i/jMi ♦^fiucHt ion program (IKP) and mke a placement decision. It is 
cofUfirtrMt ! vol y hjii^iIo to note whether timelines are ob<»erved, whether parents are 
duly notified, and whether all written docuffronts are in the student file. 
How-v<*r. a local education agency (livA) can meet all these requirements and still 
placing students in i napprc^r iate settings or failing to develop the resources 
necessary to assure education in the least rest r ict ive enviroiment. 

A growing numht^r of state and local education agencies are ready to move 
beyond siiiply looking at procedural measures of program performance and are seeking 
to prove that their programs are not just '^working^— but are effective and of 
high quality. This has led to a series of state and local efforts to develop 
evaluation procedures which measure the quality of special education programs. 

Iss ues in QMality Eroluation 

Although the issue of qual i ty evaluat ion has becorr^more important in response 
to both local program needs and national pressures on education, a nuntoer of 
factors maki* quality evaluation difficult (Farr<w» 1983). 

First, there is no consensus within either regular or special education about 
the meaning of educat ion quality. Simi larly, the approaches to evaluat ing qual ity 
are as varied as the goals by which different people attempt to <tefine it. For 
exanple, should quality be judged solely in terms of program outcomes such as 
student achievement? Are cost factors relevant to assessing quality, or should 
evaluative Judgments be made free of resource considerations? 

Second, attempts to measure program quality are carpi icated by the need to 
define their relationship to compliance monitoring. Should these efforts be 
integrated into one system, or can they exist independently? Since resources are 
I imt ted, would an enphasis on qual ity evaluat ion necessari ly involve less intensive 
cryrp I i a nee mon i t or i ng? 

Third, there is a paucity of materials designed to assess the quality of 
special education programs. Thus, the decision to measure program qual ity usually 
Inclurk^s n oorrmitment to develop appropriate evaluation materials as well. 

Fourth, any attenpt to assess progi m quality may encounter resistance from 
those whose work is being evaluated. While everyone endorses the concept of 
qual ity evaluat ion, the risk that resul ts may be unfavorable rather than favorable 
can tH* int imidat ing. 

In addition, research suggests that other factors such as the nature of SFA 
leadership, and the involvement of local "users" of the evaluation need to be 
considered and resolved to assure maximiin success of any program quality 
assessn>f»nt • 

Farrow, Hf fee t ive state mo ni tor ing policies (Qtiality monitoring ^nd 

mon itoring of st ate o perated programs) . Vtoshington, D.(". : The Center for 
the Study of Social Policy, 1983. 
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QIMLITY EVAUMTION: 
A OQNSUffiR'S GUIDB KB FOLICMMfCERS 



Education pol icymakeps ^ increasing concern with program quality is likely 
to lea'l to an increased detmnd for— an 1 receipt of-^systeimt ic evaluation 
informat ion. Many states have produced manual?; detai ling the process for obtaining 
this information, such as North Carol ir«^s S pecial Fducation Prograjn Qual i ty 
^hnuaj (North C-arolina Department of Publ ic Instruct ion, 1983); \teryland^s 
Evaluation Res ource Manual ^Maryland State [Apartment of Education, 1980); and 
Massachusetts^ Special Fducat ion Program \fanual ; A Management Tool (Massachusetts 
f>epartment of Fducat ion, 1981) • 

However, there are few resources available to assist pol ic^miakers in making 
sense of the often extensive reports ivhich result from such pr<K?esses. The 
following ''checklist" is intended to help policymakers be more thoughtful and 
effective considers of the evaluation reports they read by facilitating a clearer 
understanding of the components of an evaluation, and of the credibility of the 
reported results. The checklist outlines a series of steps which are arranged 
roughly in the order in which they appear in most evaluation reports. Each step 
includes representative activities, and provides "checkpoints "---quest ions to help 
policymakers assess each step^s activities. 



STEP 1: Reaii the ccanplete evaluation report . 

In order to assess the quality and progranimtic iiipl icat ions of an 
evaluation report, it is necessary first to become familiar with the 
program i tsel f, and the way in which it is evaluated. This is accomplished 
most "eadily through an overview reading of the report. 

Checkpoints fo r Policymakers 

• Is it clear lxhcX ly what program or program components 
the evaluation report is assessing? 

• C^n you describe the types of information contained in 
the report and in the appendices? 

• On you readily locate specific sections of the report? 

^'^^ 2' fte v^CTT the e valuftti wi mode) . 

Every evaluation has an irrplicit or explicit "model," an approach which 
defines the focus of the evaluat ion, including which questions are asked* 
Different models or approaches ask different questions, use data in 
different vvays and require different levels of time, personnel, and 
resource coomitment. Thus, the selection of an evalimtion model defines 
the scope of the evaluation, frames the interpretation of the data, and 
helps everyone understand what types of inforrmtion they can and cannot 
expect to receive^ 




A prnftTt ft r fmltc\makf'rH ndministered by the NaUunal Asuociation of Slate Boards of Education m rnoperaiinn 
tetth ih** i'tiunrd of {'hi**f SUtif^ SchiM^l Officers, Naiiorttil Conf^renve o/ Stair i.r^islaturrs, and Amenran AssfKmtion of 
Sch4h}{ Admtni$trator% 701 Fairfas St* Suite 340 Alexandria. V A 223 U (703^684 4000 



flipckpoints for Polieyiroker^ 

• ('hu you icknilify ihc particular evaluation model or 
approach wtiich is u^ed as the basis for the evaluation? 

• I«; a rationale included for its selection which clcHrly 
relates the as.sunpt ions, procedures and resource demands 
of the mode! to the assimpt ions, procedures and ava i lable 
resources of the special education program? 

• Is it clear how the evaluation model guides the program 
evaluation procedures? 

WnSP S: Hevicw the evaluation's goals aikl object ives. 

The purpose of the evaluation goals and objectives is to provicte a 
framework for ctescribing program results. Usi^lly, the objective is to 
determine how well a program is succeeding in reaching its goals. Goal 
statements are generally broad in scope and reflect long-term aims of 
the program. The evaluatof may narrwtf the goal statements to define one 
purpose or objective of the program, specify the objective in measurable 
terms, and define the timeframe within which the objective is to be 
aceompi ished. 

The goals and object ives of the evaluation provide the focus for specific 
evaluat ion quest ions. For example, theCIPPVtodel (Stuff lebeam et . al., 
1971) delineates four broad goal areas from w*iich evaluation questions 
are derived. 

1) Context Evaluation 

• What arc the unmet needs of the special education student? 

• What program obstacles or constraints exist that inpede 
meeting these needs? 

• V^at program goals or objectives are necessary to meet 
these needs? 

2) Input Kvalu at io n 

• lloes the special education program have the necessary 
prerequisite resources to deliver the needed services? 

• V/hnt strategies will be enployed to meet the program 
goals? 

• How wi n these strategies be irrplemented? 
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3) Process Evaluation 

• To w^tat extent has the program inplemenlcd act ivi lies 
viftich were (tesigned to meet program goals? 

• Wtat <k> existing program activities look like? 

4) Product Fvalufltion 

• To what extent have program goals been acconpl ished? 

• How confident can we be that observed changes are a result 
of the special education program? 

Checkpoints for Policymakers 

• Do the evaluation goals, objectives and quest ions relate 
to the evaluaition model selected? 

• Are the goals and objectives of the evaluation clearly 
I inked to the goals and object ives of the education 
program? 

• Are the evaluation objectives stated in such h way that 
you can tell if they have been aecorrpMshed? 

• Are the goals and objectives which you feel are int^ortant 
to investigate inducted? Do you have the data you want 
and need? 



Review ttw evaluation er iter ia > 

, For a particular aspect of 9 program to be considered successful, it 
must meet certain expectations. These expectations are the evaluation 
criteria. Criteria or standards can be of two general types: 1) 
qualitative or descriptive standards, such as "comnensurate with the 
student's abilities;** and 2) quantitative or nimerical stantterds, such 
as ^75 percent satisfactory*" Standards can be established through the 
use of specialists, through a review of past performance to determine 
reasonable expectations, throi^h reliance on measures of inprovement, 
or through reliance on established norms and practices^ Sometimes, the 
standards are stated in the {KJlicy under which the program was adopted^ 

Whatever standards are chosen, it is essential that they be agreed to 
by everyone* involved in the evalitfit ion, including the audience(s) for 
the evaluation report. If the standards for judging the program are not 
seen as credible and valued, the evaluation itself will not be taken 
seriously. 

Checkpoi nts for PoHcyn^kers 

• Is the type of standard to tK* used as the criterion for 
success clear for each objective? 



# Is the vNHV in which the standHrd selected eloar? 



• IX} you Mgreo with the stHfidard selected, i.e., do you 
fee I the f esu Its hs measured by this st ami^ird nrc 
believable and valued? 



STEP 5: IteviCTr the po{ Hilation and tt^ smpie seiection > 

The popuUtion is the group about v^^om the evaluator is interested in 
gathering tnformat ion ana drawing conclusions. For cxanple, a population 
might include "all students eligible for special education programs.* 
When it is fiol feasible to ineluiie the total populat ion in an evaluat ion, 
a sample must 1^ drawn t)y specifying who will be included in or excluded 
from the populat ion l^ing studied. There are many conrmonly used saopl ing 
procedure!^ (e.g», simple random sampling, stratified ranctom sampling), 
all of which help to assure that the sanple selected accurately represents 
the population t^ing studied. 

Frequently, including the entire population in an evaluation study 
involves a substantial commitment of time» personnel and money. The 
critical concern is to include enough individuals to produce a believable 
estimate of program effectiveness. 

( "heckpoints for Policymakers 

• Is the popuiat ion and/or sanple for the evalmit ion clearly 
speci f ied? 

• If a sanple is used^ is the sanpling procedure clearly 
delineated and a rationale provided? 

• Are h1 I groups which are affected by the program and wfiich 
you believe are important included in the population 
and /or sample? 

STRP 6: R eview the ev aluati on design . 

The purpose of evaluation is to provide a means of making val id decisions 
and judgmt^nts about a program^ s effect on the "^rea! world," Viewed 
broadly^ evaluation studies are t *ther quant i tat i ye » based on the 
principles of exper irwntal design (cf. CarrptK^ll and Stanley, 1963); or 
q ua! i tat ive , drawn from social science field rwthods (cf. Patton, 1978). 



yu>int ital tVG approaches assume* the necessity, ck^sirabi I ity and 
possibility of applying empirical standards to progran^ and problems 
involving hLBnan beings; they include experin^ntal and qttfisi -experimental 
tiesigns. Qualitative approaches, on the other hand, assume that human 
interactions are not all amenable to numerical and slat ist ical reduction; 
they include case studies and other natural ist ie methods which try to 
capture the meaning and successes of the program in d«*scriptive terms, 
Th«»s<* tvtfr> approaches to evaluation design are not discrete; it is not 
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ne^^essary to choose between the two. Rather, twth offer useful 
rtllerrmtives since different kinds of probl^iK and questions require 
different approaches* 

Ullimateiy, e<)nsiderat ion must also be given to what the evaluation 
audiences believe constitutes valid and reliable findings or results. 
VIeasurement and design decisions are usually made within an explicitly 
political context; they are not simply a matter of expertly selectir^ 
the "best" techniques. As Patton observed, "Design and cteta collection 
decisions are a far cry from l>clng neutral, objective, or rational; such 
decisions are political, subject ive and sat isf icing" (Patton, 1978, 202). 

C heckpoints for Policymakers 

• Is the selection of an evali^tion design clear and , 
justified by the aiire of the program, the types ^of 
informat ion needed, and the t iirofrarm; and data col- 
lect ion burden involved? 

• 1)0 you believe that the types of inforiration provided 
(e,g., statistical, descriptive) are valid and reliable 
data upon which you would be willing to take action? 

snap 7: Heview the chta coUectSfWi and analyses pfaec3scs . 

It is possible to coliect evaluation information in a variety of wys, 
such as questionnaires, observations, interview, rating scales, 
dociOTcnt review, and perforrrance tests. Care should l» taken so that 
the method or methods (rtiosen enhance the reliability and validity of 
both the data collected and the total evaluation design. 

Once evaluation information has been collected, dbtr analysis can be 
conducted^ Most LFA and SFA procedures rely primarfy on descriptive 
statistics such as frequencies, percentages and means • Quotations, 
illustrations and (Ascriptions may be included to add to ♦^e readability 
and bel ievabi I ity of the statistical cteta* One of the most iin>ortant 
standards for cteta analysis is, "Is the evaluation inforrmtion presented 
in the most clear and useful wy possible?" 

CTheckpo ints for Pol icymakers 

• Does the evalimtion address the extent to which the 
instruments measure what they are supposed to measure 
(i.e., are they valid)? 

• Does the instrument design give you confidence that if 
t he i ns t rimient s were adtni n i st ered more than once to 
similar groups (or to the same group) they would yield 
consistent results (i*e., are they reliable)? 

• Do the ways in which the data are analysed and presented 
reflect the program objectives and make sense to you? 
Are the types of data provided (e.g., percent correct, 
percentile rank, grade equivalents) of use to you? 

iO 



STRP 8: ilevi cw the overall ewilwitiwi tepofX . 

The evniuat ion report should be easy to understand, and should eormiuni cat e 
v^fhat (jtone, how ft was aeconnpl ished, and why, in a wy which is 
credible to you as a primary evaluation audience . In adaitic/ii, the 
evaluation report ought to be timely for your purposes and should focus 
at least in part, on your identified information needs. The question, 
'^does evaluat ion make a difference?" is tcK> often answered in the negat ive 
because of a failure to gear the evaluation rejwrt tcmard act ion by 
decisionmakers. 

Checkpoints for PoUcyrakers 

Format Oonslcterat ions: 

• Is there an Executive Swinary which provictes a clear 
description of the evaluation pr<Hiedures and results? 

• Is the report arranged in such a way (e.g. , with 
appropriate tables and section headings) that you can 
identify what information is of interest to you and find 
it easily? 

Cbntent Considerations: 

• Does the repoi't provide believable evidence that the 
positive results r^rted <K;cufred as the result of the 
special education prt^ramor are other explanations, such 
as normal growth of the students, just as plausible? Are 
things better than what wuld be predicted without a 
special education program? 

• Are alternative interpretations of the (teta presented, 
and the reasons for their rejection macte clear? 

• Are the reported results educationally significant, that 
is, are they ''of nontriyial magnitude , in a content area 
generally accepted as important^ whTch can be achieved 
at a reasonable cost " (Tallmac^e, 1977, 34)? 

• Are the data presented consistent, e.g., do the nur^rs 
in the text and tables agree? Are the inferences drawn 
from the dbta consistent with the evidtence, e.g., <toes 
the evaluation claim educatioral significance only when 
the data sup{K>rt it? Are claims of causal ity 
substantiated by the evidence? 

• Are there any "unanticipated consequences" of the 
progran? Are they ctoeumented and explained in a v«5y that 
makes their impact clear? 
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STEP 9: Pes i pi a program iaprowaent Mttnagaaent plan . 



The valt^ of an evaluation^ especially one which addresses Issues of 
program qi^lity, rests in the usefulness of its recomnendat ions. 
Reconmendat ions cover tw> broad areas: 1) reiwdiation of ^fieit areas; 
and 2) inprovonent of activities determined to be of primary im(H>rtarice» 
These require both an object ive analys is of evaliat ion data and a 
subjective analysis of participant concerns^ From such analyses, 
priorities for action can be selected, and a n«nagefnent plan for 
accomplishing these priorities can be generated* 

Checkpoints for Policymakers 

• Are the recomtnendat ions clearly based on the evaluation 
snformaticMi presented? Do you believe these 
ret^omnencfet ions effectively address the apprc^r iate 
progrofi strengths and waknesses? Is it clear trtio has 
responsibility for taking action on the recomnendat ions? 

• Do the policy inpl icat ions and reconmendat ions folloi^ 
directly from the data? Does the report provide the 
rationale and information you need to act on these 
recommendations? 



Cbneltwiwi 

The preceding simmEiry of program evaluation procedures illustrates 
representative components of a technically adequate evaluation. While these 
components can be easily deltr^ted, in real life many factors limit their 
ifiplementat ion,. These include diversity in progr^ objectives, heterogeneity of 
program staff, and swings in resource allocation as wll as timelines, and 
pol it ics. 

Policymakers need to consider both the standards for a qt^lity evaluation, 
and the constraints involved in c^erat ional izing such stan<terds. The key is to 
be involved and inf ormed at all steps during the evaluation process! 
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mXWm EVAUMTION 



[.1ST 



St ep 1: HoHd ihe com plete evaluation report , 

• Is it clear exactly what program or program eonponents the evaluation 
report is assessing? 

• Can you de^^cribe the types of information contained in the report and 
in the af^endices? 

• Can you readily locate specific sections of the report? 

Step 2 : Review the eval^iation model . 

• Can you identify the particular evaluation model or appr<^ch w*iich is 
used as the tmsis for the evaluation? 

• Is a rationale inducted for its selection which clearly relates the 
asswipt ions and procedures of the model to the assiarpt ions and procedures 
of the special education program? 

• Is it clear how the evaluation model guides the program evaluation 
procedures? 

Step 3 : Review the evaluation's g(»ls and objectives ^ 

• Do the evaluat ion goals, objectives and quest ions relate to the evaluation 
model selected? 

• Are the goals and objectives of the evaiimtion clearly linked to the 
goals and objectives of the special education program? 

• Are the evaluation objectives stated in such a way that you can tell if 
they have been acconnplished? 

• Are the goals and objectives which you feel are in^wrtant to investigate 
included? Do you have the data you \mnt and need? 

Step 4: Review the evaluation criteria . 

• Is the type of standard to t>e used as the criterion for success clear 
for each objective? 

• Is the way in which the stan<terd wiS selected clear? 

• Do you agree with the standard select ed» i.e., do you feel the results 
as measured by this standard are believable and valued? 

StegJ: Review the {HH>ulation and sample selection . 



Step 6: 



!s the population and/or sample for the evaluation clearly specified? 
If a sample is used, is the sampling procedure clearly delineated and a 
ratioiwle provided? 

Are all groups \^ich are affected by the program and ^ich you believe 
are important inducted in the population and/or sanple? 

Review the evaluation ctesign . 
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Is the selection of an evaiimtion design clear and ju tif led by the aims 
of the program^ types of data needed and the t imeframe and data collection 
burden involved? 

• Do you believe that the types of informat ion provided (e.g., statistical, 
descript ive) are val id and reliable data upon which you would be willii^ 
to take act ion? 

A project fcr poiicymakerH admtnUiered by the National AaaociatioD of Stale ]^>ards of Education in cooperaiUm 
wtik ihe Counctt of Chief State SrAno/ Officers, National Conference of State Legislatures, and American Assix^iation of 
^School AdminiBtratort 701 N. Fairfax Suite 340 Alexandria. VA 22314 {703^-4000 
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^tpp 7: Review the data <?onection and analyses proces ses, 

• lk>es the evnliintion address the extent to which the instruments measure 
i^at they are supposed to mensure (i,e., Hre they valid)? 

• Dae^ the instrument design give you confidence that if the instrianents 
wre artninistered more than once to similar groups (or to the samt^ group) 
they would yield consistent results (i.e^, are they reliable)? 

• Do the wiys in which the data are analysed and presented reflect the 
program objectives and make sense to you? Are the types of data provided 
(e.g., percent correct ♦ percentile rank, grade equivalents ) of use to you? 

^tep 8 : Review the overall evaluation repo rt. 

Format Considerations 

• Is there an Executive SiBrmary which provides a clear description of the 
evaluation prcK*edures and results? 

• Is the report arranged in such a way (e.g., with appropriate tables and 
section headings) that you can identify vrfiat information is of interest 
to you and find it easily? 

Content Considerations 

• Does the report provide believable evidence that the positive results 
reported occurred as the result of the special education program or are 
other explanations, such as normal growth of the students, just as 
plausible? Are things better than what would be predicted without a 
special education program? 

• Are alternative interpretations of the data presented, .and the reasons 
for their rejection made clear? 

• Are the reported results educationally significant, i.e., are they '^of 
nontrivial magnitude , i n a content area general ly accepted as important , 
which can be achieved at a reasonable eost ^? 

t \re the data presented consistent, e.g., do the niflTd)ers in the text and 
tables agree? Are the inferences drawn from the data consistent with 
the evidence, e.g., does the evaluation claim educat ional significance 
only when the data support it? Arc claims of causality substantiated 
by the evidence? 

• Are there any ^Minant icipated consequences" of the program? Are they 
documented and explained in a way that makes their inpact clear? 

^ tep 9 : Design a program inprove fT^nt TOnagcment^ plan . 

• *\re the recommendations clearly based on the evaluation information 
presented? Do you believe these recofrmendat ions effectively address the 
appropriate program strengths and weakness? Is it clear who has 
responsibility for taking action on the recormx?n<tet ions? 

• Do the policy implications and reconroendat ions follow directly from the 
data? Does the report provide the rationale and inforcmtion you need 
to act on these recornm^mlat ions? 
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A FQLfCttMCBR'S EVAIiWriGN aXS^MiOr 



KvalUHtion, like many other fields, has a language utiich deserites its 
processes and products in a special i zed manner » The following glossary is provided 
to clarify some of the basic evaluation research terminology ♦ (Katzer et.al., 
1982). 

Act ion research : Research in wJiich the decisiormaker using the results of the 
research takes part in the research. 

Appl ied research : Research carried out to discover new knowlec^e that has iime- 
diate applicability. 

Average : A measure of central tendency (the middle) of a set of scores. Usually, 
Tt is synonymous with the mean. 

C ane study; An analysis and explication of a single situation or single case 
TuF.it of analyins; subject), often using qiml i tat ive methods. 

Before an d after design ; An experimental design in irfiich subjects are measured 
on a given variable before and after the inctepenctent variable (or treatment) 
is adninistered. 



Causal i ty : The concept that some phenomena cause other phenomew* 

Control group; In a study made up of several different treatment groups, the 
control group is the one against which the others are comp^ired. Often, the 
control group (K>es not receive any treatment. (Also known as the con|>ari$on 
group. ) 

Correlat ion : In informal usage, correlation is often used as a synonj^ for 
''associat ion, ^» although there is a technical distinction between these terms. 
Correlation is also used to stand for a particular measure of association, 
Pearson's r. 

Demographic variable : An attribute variable of a person; usually a variable 
that classifies a person into social groupings. Cdmmon d^nographic variables 
arc age, sex, social class, ethnicity and marital status. 

Dependent variable : The variable that is thought to be affected by an 
Tni^pen^nt variable (Also knwn as the criterion, outcome, predicted or 
response variable.) 

Descriptive statistics : Statistics that simrmrize a set of cteta; inferences 
from the data to a larger population are not me 

Frrpi r ica! : Subject to critical evaluation through observation or experiment 
leather than through speculation or theorizing. 

Kxperimental group : A treatment group that receives a level of the inctepenctent 
variable (or treatment ) vrfitch is of substantive interest, as contrasted with 
the control group. 



A prajf^rt fvr pfihrymakern administered by the National Association of State Boards of Education in cooperation 
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F.xlernHl VBlidity; The generality of research results. 

KxperimentHl research : Research in \ti\ich the subjects have been subjected to a 
treatment and the purpose of the study is to assess the effect of that 
treatment or event. 

Field experiment study : Non-experimental research conducted in a non- 
laboratory setting, usually the natural enviroranent of the phencjmenon under 
study. 

For mative evaluation : Evaluation which is conducted during a program or 
process, usually for the purpose of pinpointing progress /problems and 
altering the process, if necessary, while it goes on. 

Independence : No association or relationship. 

I ndependf H t var i"ble : The variable thought to produce a result on the dependent 
var table. 

Internal val idity : A consideration of whether the independent variable produced 
variation in the dependent variable. Thus, internal validity is a type of 
factual accuracy. 

!iOngitudinal study : A study conducted over a period of time, although sometimes 
only at a few discreet points in time during that period. 

Model ; A sinplified description of a phenomenon, often in words, pictures or 
syrotx>Is. 

Naturalistic observation : A method of gathering data that involves making a 
detailed record of events as they occur in their natural setting v^ile having 
as little effect as jwssible on those events. 

Non-experimental design; A research design in which it is impossible to 
control all biases and protect internal validity. 

» 

PHrftdigm: A complex theoretical moctel that is used to explain phencxnenon. 

Pol icy research ; Research conducted to help formulate or evaluate policy* 

Po pul«t ton : All possible observations or units which could be used in a study* 

Post -test : Vfeasurenent of experimental subjects on a given variable after a 
treatment is given. 

Prr -test : Measurement of subjects on a given variable before the independent 
variable/treatment is acfcninistered. 

Qualitative research : Research methods that attempt to describe and understand 
people or social entities from their own point of view. 

Q uant i tat ive research : Research methods that attempt to categorize and 
sunmarize observations numerically. 

Qtia si - exp erimental research : An experiment in which some, but not alU sources 
of potential bias are under the control of the experimenter; at least ranckm 
assigmient of subjects Is usually missing. 
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Rrj i«t>i 1 ity: Fxtent to which a fwasuring instrnnent vwould give the sarw value 
rr use«l <>v<*r Hfid over providing the attribute measured did not change. 

Research design : The strategy or plan of an experiment » often focusing on the 
control of possible biases and the schedul ing of treatments and measurement 

Samp le: The subset of the population actually used in the research. 

Stat ist ical signi f icance: The outcome of significance tests In which the 
results are shown to have a low protobility of being due to chance alone, 
thereby eliminating chance as a viable cause of the results, 

Su miative evaluation : Fvaluatlon v^>h is concerned with determining overall 
effectiveness of a completed program or process, usually for the purpose of 
deciding whether or not to repeat it» or what changes^ if any, need to be 
nwde before repeating. 

Val idity : How vw?l! an instrument measures the phenomenon under investigation. 

Katzer. J., Cook, K., flc Crouch, W. Evaluating infornBtion; A guide for 

users of social science research (2nd ed. ). Massachusetts: Add i son-Wesley, 
19a2, 



SEA EFKKTS TO EVAIiKTE THE QtWUm OP SPHCIAL EDUCATION FROGRM 

rX'spjif thV f»rmi(lHble chftllenges involved in assessing program qua 1 ity, a 
number of skNs tutvc eonmitted themselves to "program qua I i ty evaluation," efforts 
which include « wide variety of methods, and which represent a multiplicity of 
views about the purpose of qual ity monitoring. Tflree such efforts are siBtmar ized 
in the following section. For a more detailed analysis of these and other state 
evrtlUHtion efforts, see Farrow (1983), and fiXSDF (1984). 



Webraska Departiaent of Education's System of Program EffectiTOness Ewlufttion 

The Nebraska Department of F.ducation (NDR), partly in response to state 
statute (Nebraska <*tatute No. 43650), is developing a system of program qual i ty 
evaluation (PQK) which is designed to meet both the SFAs' and LFAs' needs for 
evaluative information. In 1981, a 20-mcmber Program Effect iveness Development 
roninittee developed a set of six standards or what thfey termed "service foals" 
rwspd on their perceptions of the character ist ics of an effective special edufjation 
program. The six major topic areas are: 

1. arkninistration, relating to the inlernal operation of the district; 

2. service delivery, relating to the identification of students and the 
provision of services; 

3. curriculum/materials and facilities; 

4. eommunicBt ion with eonsimiers; 

5. fiscal aspects; and 

6. system aspects, relating to the relationship t>etween the special 
education progranw and other prograns or agencies which may be affected 
by handicapped students. 

The rogram Quality Evaluation model was field tested in 20 districts during 
the 1983-84 school year. Eventually, NDE plans to Implement the system statewi<te. 

For more information, contact: Mr. Don Anderson, Compl iance Director, Special 
f-ducation. Box 94987, 301 entennial Mall South, Lincoln. NE 68509-4987; (402) 
471-2471. 

Missouri Department of Elementary and SeeomiBry Education's System for Program 
Rvaliwtion 

The Missouri Department of Elementary and SeconAjry Education is developing 
a Special Education Evaluation (SEE) model to be used on a voluntary basis by LFAs 
«s « vwiy to document the effect iveness of their programs. The SEE model Identifies 
sixmajor corrponents that address the quality of local special education programs: 



A prvjevt fi r fmlu ymukrrs administered by the National Association of State Boards of Education in cooperation 
with the t'ounctl of Chief State SrW Officers, National Conference of State Ugislatures, and American Association of 
School Administrntors 70| N, Fairfax Su, Suite 340 Alexandria, VA 22314 (703>684-4000 



K irlt^nf if icHit ion of hRndjt»Hpped children; 

2. evaluflt ion/diHpnos is op reevBluat ion; 

3. development of indi vidual ixed education programs (IFPs); 

4. plr'iec?ment ; 

5. in^)len)rnrHt ion of IHPh; and 

8. annual program review. 

For each component, the *?KF. model looks at resources, services, outcomes and 
feedtmck in order to view the program as a whole. The SFR field tested during 
the acadctnic year. The system will be inf^jlemented sef^rately from the 

slHtp*s compliance monitoring system. 

For more informtion^ eontaet: Wr. TedNickell, State of Missour 1 , Department 
of FlHuentHrv and Secondary Education, P.O. Box 480, Jefferson City, 65102; 
(314) 751-2965. 

* « jH « « « 

Worth Qarol ilia Departtnent of Pubiie Instruction's Progfttm Qmlity Evaluation 

North Carolina's Program Qua! ity Fvaluation (PQR) system is designed to 
provide f.FAs with a mechanism for determining program quality. PQF. includes goals 
in three main areas: 

1. determining learner gains /out comes; 

2. lcx"»t ing and evaluating learners; and 

3. placing learners appropriately. 

For each of these areas, PQF establ ishes program object i ves and speci f ic evaluat ion 
tjUrstioMs. i>:as can srl the numerical standard for satisfactory perforfTOnee at 
different levels, reflecting varying expectations for local prograrre. 

The P^K WIS field tested in 16 LFAs during the l983'-84 academic year. LFAs 
wnre cofT^iet i I i veiy selected through an incentive grant program i^ich provided up 
to $10,000 to assist with inplementat ion of the evaluation. Eventually, North 
Oirol ina plans to confine these evaluat ion act ivit ies with the on-going conpl iance 
monitoring procedures. 

For more informtion eontaet: The Division for Exceptional Children, North 
(%iroIina DrpartnTK^nt of Public Instruction, Kducat ion But Iding, Raleigh, NT 27601; 
(miU 733-3921. 
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The chHft •)«! lh»' following page highlights somf mior charHCter i st ics of the 
three slate exHfr|»ios de«^i*r it>rd in this •.ection. Perhaps the most striking 
observation that e«n be mnde nbout these and other efforts to develop and inplement 
program qiwlity evaluation is that "the methods used by SEAs to measure qua 1 ity 
are varied and represent divergent views about the goals of quality monitoring" 
(Farrow, 1983, 30). In special education, as i-n many other fields, quality means 
different things to different people. 

»♦♦♦♦♦♦♦♦♦»»»♦•♦♦♦♦♦<♦♦♦*♦♦••♦♦♦•*♦**♦♦♦**«♦♦***♦*♦♦♦*♦*♦♦♦♦♦♦♦♦*♦♦♦♦*♦♦♦*♦♦♦♦ 

Farrow, F, Effective state tnonitoring f>oligies (Quality monitoring and 

monitorfng of state operated programs) . Washington, D.C.: The Center for 
the Study of Social Policy, 1983. 

National \ssociation of State Directors of Special Education (NASDF.). State 

responses to the ''Annual Evaluation^ requirernents for state plans . 

Wishin^ton, D.C. : Author, 1984. 

♦♦♦♦*♦♦•♦•••»»•♦*♦•♦•♦♦♦»♦♦♦♦♦♦♦♦♦»♦♦♦♦♦♦♦♦♦♦♦♦•♦*••♦•♦**•♦•*•*♦♦♦♦•*♦♦♦♦♦♦♦♦♦ 
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